Resilient Distributed Data Management Protocols in Dynamic Resource Environments

Traditional cloud systems are designed to tolerate server or rack-level failures that are uncorrelated and unpredictable. Such systems successfully deliver highly-available cloud services at global scale. However, the increasing criticality of cloud services to the overall world economy is causing concerns about the impact of power/network outages, cyber-attacks, administration errors, or other causes of datacenter or larger-scale failures on cloud availability. Recent experience shows that these events can trigger cascading failures and global-scale service outages. Moreover, the continued drive for resource use optimization also increases resource dynamism and volatility. We seek to understand the resilience problem of existing cloud systems, and create highly available data services despite daily, concurrent datacenter outages. We begin by studying the impact of concurrent datacenter outages on data service availability. We characterize the failure behavior of distributed protocols that are widely used in Cassandra [35], varying protocol configurations and resource properties. Our study reveals that using such protocols to achieve high availability under correlated, datacenter-scale outages are costly in storage and update traffic, requiring replication factors of 10 or more. Further analysis reveals that such limitation arises from inflexible replication and management strategies. We argue that such resilience problem is due to the traditional assumptions of resource failures that are no longer applicable to today’s cloud computing environments. While traditional systems assume failures to be at server or rack level and with uniform, fixed failure rates, the emerging failures we are currently facing are highly correlated, varied across datacenter resources, and with more dynamism and higher frequency. To deliver highly available data services in the face of frequent, correlated outages, we propose new protocols that take the novel approach of exploiting statistical models for datacenter resource failures. The complete modeling of failure statistics enables us to (1) exploit the failure correlation for efficient load management and data storage/movement overhead upon largescale outages, (2) reflect the differences in resource reliability across datacenters for intelligent replica placement, and (3) efficiently respond to failures and recoveries with statistics-based prediction and awareness of correlated failure domains. More specifically, our protocols adopt statistical-model-based management strategies, including dynamic replica management and dynamic quorum that can adapt to real-time resource availability changes, as well as asymmetric resource treatment. In this way, our protocols can achieve 99% service availability and high consistency with low fixed cost and outage-proportional overhead. To demonstrate the improvement of our protocols, we will prototype and implement them in the framework of Cassandra. The design, the implementation, and the evaluation of those protocols will be included in the full thesis.